Telehealth: The Need for Evaluation Redux

The paper in this issue by Hripcsak and Wilcox, " Reference Standards, Judges, and Comparison Subjects: Roles for Experts in Evaluating System Performance, " 1 is well written and presents a thoughtful analysis of the topic. As the authors acknowledge, however, there is more to the evaluation of clinical informatics systems than can be accomplished through comparison to experts. 2,3 Hripcsak and Wilcox focus on " how to use experts in evaluating systems when one needs them, " whereas this commentary focuses on the question, " when should one use experts as part of a system's evaluation " The two perspectives are complementary rather than contradictory. As noted previously, 4 System evaluation in biomedical informatics should take place as an ongoing, strategically planned process, not as a single event or small number of episodes. Complex software systems and accepted medical practices both evolve rapidly, so evaluators and readers of evaluations face moving targets. … [C]urrent thinking recognizes that such systems are of value only when they help users to solve users' problems. Users, not systems, characterize and solve clinical diagnostic problems. The ultimate unit of evaluation should be whether the user plus the system is better than the unaided user with respect to a specified task or problem. … If the ultimate evaluation of a system depends on whether users of the system perform a specified task better when they use the system than when they don't, then there must be public, objective criteria (a " gold standard ") made available before an evaluation begins, to determine the quality of performance of an individual on a task (independent of whether the individual uses a decision-support tool). Hripcsak and Wilcox state that experts can be used in three evaluation settings: to generate, through intro-spection and expertise, the reference standard per se (e.g., by providing a list of " correct " diagnoses or of " correct " therapeutic interventions based on a reading of the problem at hand); to judge (and label) the individual behaviors of subjects in the study—on a scale ranging from " optimal " through " acceptable " to " inadequate " —without providing an absolute list of " correct answers " ; and as actual subjects in the study, to make it possible to rate how well the system performs in comparison with the performance of human experts. Hripcsak and Wilcox's first two scenarios assume that no absolute, …